Knowledge distillation via instance-level sequence learning

Recently, distillation approaches for extracting general knowledge from a teacher network to guide student have been suggested. Most existing methods transfer the by feeding sequence of random minibatches sampled uniformly data. We argue that, instead, compact should be guided gradually using samples ordered in meaningful sequence. Thus, gap feature representation between and can bridged step step. In this paper, we provide curriculum learning framework via instance-level learning. It employs early epoch as snapshot create network’s next training phase. performed extensive experiments CIFAR-10, CIFAR-100, SVHN, CINIC-10 datasets. When compared with several state-of-the-art methods, our achieved best performance fewer iterations.